DOCUMENT RESUME 



ED 391 507 



IR 017 660 



AUTHOR 

TITLE 

PUB DATE 
NOTE 



PUB TYPE 



EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



Gibbs, William J. 

Computer-based Testing and Strategies for Distance 
Learning. 

[95] 

7p* ; In: Eyes on the Future: Converging Images, 
Ideas, and Instruction. Selected Readings from the 
Annual Conference of the International Visual 
Literacy Association (27th, Chicago, IL, October 
18“22, 1995); see IR 017 629. Figures may not 
reproduce wel 1 . 

Reports - Descriptive (141) — Speeches/Conference 
Papers (150) 

MFOl/PCOl Plus Postage. 

^Computer Assisted Testing; ^Computer Software 
Development; Distance Education; Educational 
Technology; Feedback; ^Programmed Instructional 
Materials; Student Reaction; ^Teacher Developed 
Materials ; Teacher Education; *Test Construction 
*Test Analysis 



ABSTRACT 

TestMaker is a project in which computer-based 
programs are being developed to help educators create tests. The 
program was designed as an instructional and developmental tool for 
teacher-education students. TestMaker consists of four modules: 
Advisement; Test Creation; Student Test; and Presentation Analysis. 
The Advisement module runs concurrently with the Test Creation module 
and provides users with system-generated advice and user-selected 
advice related to effective test construction. All question items, 
correct answers, alternatives, and feedback created in the Test 
Creation module are presented in the Student Test module; during 
testing, student responses, elaboration and instructional time are 
recorded in a file which can later be reviewed by the instructor. The 
Presentation Analysis module was designed as a presentation tool for 
distance learning; it assists in the analysis of questions by 
allowing the instructor to log the number of students agreeing and 
disagreeing with each question alternative. Future developments with 
the TestMaker project will investigate the possibility of testing 
students at remote distance learning sites. (Contains 11 references.) 
(AEF) 
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Introduction 

The assimilation of computer 
technology in society has enabled 
computers to be used for a variety of 
educational purposes with some standard 
uses being in the area of learner 
assessment (Eaves & Smith. 1986). 
While paper-and-pencil tests have been a 
standard means of educational 
measurement (Olsen, Maynes, Slawson. 
& Ho, 1989), assessment methods are 
changing and will continue to change with 
the development of more advanced 
computer-based learning environments. 

In terms of educational 
measurement, computers serve many 
functions and offer some distinct 
advantages over traditional methods. From 
an administrative perspective, computers 
provide an accurate experimental tool whh 
which to conduct and manage learner 
asses.sment. They can control many of the 
routine ta.sks or clerical duties associated 
with such processes. Computers increase 
efficiency in test duplication, 
administration, scoring, item banking, 
record keeping and analysis (Eaves and 
Smith, 1986) and offer more organized 
assessment (Johnson and Harlow, 1989). 

From an in.structional perspective, 
computers enable standardized testing 
conditions, individually administered te.sts, 
immediate test scoring, increased variety 
of testing formats, and ability to collect te.st 
and item latency information (Olsen, 
Maynes, Slawson, and Ho, 1989). They 



are capable of presenting test items and 
evaluating them. For example, man\ 
computer testing systems generate unique 
question sets for each learner. Based on 
previous responses to questions, the 
computer determines which questions are 
appropriate for each individual (Cudeck. 
Cliff and Kehoe, 1977). Computer-based 
tests give educators the capability of 
providing learners immediate feedback or 
knowledge of results. With paper-and- 
pencil tests, there is often a considerable 
time lapse between the time of te.sting and 
the time at which test results are reported. 
Finally, computers have the advantage ot 
being completely objective, which is not 
possible with humans (Johnson and 
Harlow, 1989). 

Computer-Based Testing 

The research literature appears to 
support the claim that computer and paper- 
and-pencil test administration produce 
comparable results in learner achievement 
scores. Stile and Pettibone (1983), for 
example, found no significant differences 
in acquired scores when comparing 
computer testing to paper-and-pencil. 
Eaves and Smith (1986) report that 
undergraduate college students' 
achievement scores on computer 
administered tests are similar to that of 
paper-and-pencil. 

Assessment is a fundamental part 
of education. Traditionally, multiple- 
choice and true/false tests have been the 
predominant means of assessment. While 
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used heavily, aivi appropriate in many 
cases, current educational perspectives 
(e.g.. constructivists approaches) require 
thal assessment also focus on higher order 
thinking and problem-solving abilities. 

A potential advantage of 
computerized testing is the capability to 
allow users to interact with tests. 
Research suggests that allowing for 
interaction during testing promotes 
learning. For example, giving learners the 
opportunity to respond to feedback has 
been shown to facilitate higher levels of 
achievement (Marrone, 1991). Activities 
during computerized testing which help 
learners relate new information to existing 
knowledge (e.g., responding to feedback) 
may prove beneficial to learning and 
should be considered by instructional 
designers. It is possible that as- 
computerized testing procedures become 
more interactive and dynamic, higher order 
learning may be facilitated. 

Purpose of Paper 

This paper reviews the TestMaker 
project in which computer-based programs 
are being developed to help educators 
create tests. Interactive testing, record 
keeping and advisement on test creation 
are discussed. Also presented is a 
discussion of a computer-based module 
developed for this project, which is being 
used for distance learning instruction. The 
module provides for presentation and 
analysis of question items and feedback 
during class lectures. 

TestMaker Objectives 

The development of TestMaker 
was initiated by requests from faculty at 
Eastern Illinois University for a computer- 
based test development and learning tool 
that would 1) provide instruction on 
effective strategies for creating tests and 2) 
assist in the creation of computerized tests 
that evaluate and record student 
performance. The program was designed 
as an instructional and developmental tool 
for teacher-education students. Recently, 
the program was expanded to include a 
Presentation Analysis module which 



enables educators during their class 
lectures to present questions to the class, 
ask for feedback and graph class 
responses. The objectives of this project 
were to; 

1 . create a test development module that 
provides students instruction and help with 
creating tests, 

2. create a computerized test development 
tool that facilitates easy test creation to 
meet an expressed need of faculty, 

3. develop a computer-based module that 
incorporates testing strategies aimed at 
facilitating higher order learning, 

4. develop a computer-based module to be 
used for distance learning instruction that 
enables instructors to review and analyze 
question items and feedback during class 
lectures. 

Overview of Prototype Design 

TestMaker is a prototype that 
consists of four modules; 1) Advisement, 
2) Test Creation, 3) Student Test, and 4) 
Presentation Analysis. Currently, the Test 
Creation module enables users to create 
multiple-choice tests. From a question 
template screen, test items and feedback 
are entered and subsequently a student test 
file is created. The Advisement module 
runs concurrently with the Test Creation 
module. It provides advice and 
information on such topics related to 
effective test construction. At any point 
during test creation, users have the option 
to execute the Advisement module. Once 
all test items are entered, a test is created. 
This allows students sitting at a computer 
to take the test, interact with it and receive 
feedback on their performance. The 
Presentation Analysis module assists in the 
analysis of test items for class lectures and 
test reviews. Each module is discussed in 
more detail below. 
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Modules of TestMaker 

Advisement Module 

The Advisement module provides 
users with two types of advisement: 
system-generated advice and user-selected 
advice. As tests are being created, the 
system will prompt users when it appears 
that an item violates test construction 
guidelines. This is referred to as system- 
generated advice. For example, if a user 
attempts to enter twelve alternatives for a 
multiple choice question, the system will 
advise the user that this may be 
inappropriate. 

User-selected advice operates 
concurrently with the test creation 
program. When selected, this option 
presents a scrolling list of topics. With the 
computer's mouse, users click on key 
words in the list and are presented a 
window of information on the topic. The 
Advisement module provides information 
on such topics as types of feedback, 
designing feedback for intellectual skills 
instruction, test item length and 
complexity, hints for writing test items, 
and test validity (see Figure 1). 



Figure 1 

TestMaker: Advisement Module 
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Test Creation Module 

To create a multiple choice test, 
users enter each question and a set of 
alternatives by typing in pre-defined text 
blocks. Once questions and alternatives 
have been entered, the system prompts the 
user to choose a correct answer. This is 



done by clicking on the correct answer for 
a particular question. Previously entered 
questions and alternatives can be viewed 
and edited in the system's edit mode. This 
mode allows for each question to be 
modified. Additionally, all test items can 
be viewed together and altered as needed. 

For each question created with the 
module, users can incorporate feedback 
which will be displayed for students 
during testing. Numerous studies indicate 
that learning is effected by practice and 
feedback. Giving learners feedback as to 
their progress in a particular subject is an 
essential ingredient of effective instruction. 
Consistent feedback helps to motivate 
learners in learning pursuits; it gives them 
knowledge of their performance and it 
enables the instructor to assess progress 
made by individuals (Corcoran, 1985). 
Types of feedback range from merely 
presenting results to presenting in-depth 
explanations about responses (Lee- 
Sammons & Wollen, 1989). 

As a form of feedback, the correct 
answer can be presented for incorrect 
responses. Thus, if the instructor enters 
feedback and selects the correct answer 
display option, the student, upon 
responding incorrectly to a test item, 
would be presented the correct answer 
along with the instructor's feedback. 
Providing students knowledge of correct 
responses may be a useful strategy. The 
process of comparing and judging 
responses may serve a as means to 
increase interaction and to subsequently 
invoke deeper cognitive processing. It ma\’ 
encourage learners to think about their 
own work which can add substantially to 
assessments (Tittle, 1991). 

The type of information processing 
in which learners engage during 
instruction impacts the degree of learning 
that takes place. Therefore, a critical factor 
in designing learner respon.ses is the level 
of processing they invoke in learners 
(Jonassen & Hannum, 1987). The test 
creation module has an option for 
interaction during testing. While still under 
development, the purpo.se of this feature is 
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to allow test takers to respond to feedback, 
explain their answers and/or to justify why 
they are right or wrong. For example, if 
the instructor had selected this option, 
students, upon responding to a particular 
test item, would be prompted to elaborate 
as to why their response was correct. The 
student would then type an explanation 
which could later be reviewed by the 
instructor. This type of elaboration is 
hypothesized to facilitate information recall 
by providing additional information for the 
formulation of responses (Gagne, 1985). 
Activities such as elaboration may help 
learners organize information and thus 
assist them with integrating new 
information with their existing knowledge 
(Park & Hannafin, 1993). In this respect, 
the testing experience itself becomes a type 
of learning strategy which helps to 
facilitate learning. 

Student Test Module 

The test file is generated from the 
test creation module. Once generated, it 
can be given to students on disk or over a 
local area network. All question items, 
correct answers, alternatives and feedback 
created in the Test Creation module are 
presented. During testing, student 
responses, elaboration and instructional 
time are recorded to a file which can later 
be reviewed by the instructor. Upon 
completion, students are given a grade. 

Presentation Analysis Module 

The Presentation Analysis module 
was designed as a presentation tool for 
distance learning. It resulted from an 
instructor's need to present, analyze and 
discuss multiple choice questions during 
class lectures. In reviewing a particular 
class topic, the instructor would present a 
question to students and have them discuss 
it to determine the correct answer. This 
was originally dons with paper-and-pencil. 
However, when designing class materials 
for distance learning, the instructor saw a 
need to provide a more thorough analysis 
of questions and thus the Presentation 
Analysis module was developed. 



Since the program is used for 
distance learning, much consideration had 
to be given to visual design. In a distance 
learning environment, a scan converter is 
used to convert the computer's output to 
an analog signal that the distance learning 
system could recognize. Typically, in this 
conversion process, some degradation in 
image quality results. Moreover, distance 
learning systems require that attention be 
given to the design of visual materials. 
Therefore, font sizes should be sufficiently 
large enough in order for the visual to be 
read easily. A font of 24 points is often 
suggested as an adequate size. However, 
multiple choice questions when entered 
into the module at a 24 point font size tend 
to clutter the screen. Depending on the 
amount of text in the question being 
displayed, words were often outside 
viewers' visual field. To circumvent this 
problem, text size was made adjustable. 
Therefore, any word on screen, if clicked, 
would enlarge to a bold 24 point font size 
and subsequently reduce to a non-bold 18 
point font size when clicked for a second 
time. This served two purposes. First, it 
helped to ensure that text could be viewed. 
Second, adjusting font sizes served as a 
cueing device to direct viewers' attention 
to the specific points being discussed. 

This module assists in the analysis 
of questions by allowing the instructor to 
log the number of students agreeing and 
disagreeing with each alternative. For 
example, the instructor can highlight a 
particular question alternative and ask 
students if they agree or disagree as to 
whether it is correct. The number of 
students agreeing or disagreeing can be 
entered and dynamically graphed for 
students to view at the remote sites. 

The module keeps a running tally 
sheet of the number of students agreeing 
and disagreeing with each question 
alternative. This is useful for determining 
the concepts students understand and those 
which they do not understand. The 
number of questions and the sequence in 
which they are presented is tracked by the 
system. For instance, an instructor may 
choose to review questions out of order. 
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At any point during a lecture, the instructor 
can view the tally sheet and determine 
those questions which were viewed and 
those that were not. Figure 2 depicts the 
analysis mode of the module. 

Figure 2 

Presentation Analysis Module 
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Each question alternative can have 
feedback associated with it explaining why 
it is correct or incorrect. By selecting the 
Feedback Option and then clicking any 
alternative, feedback for that item is 
displayed. All questions and feedback can 
be easily updated and new items can be 
imported from a word processor. 

The Presentation Analysis and Test 
Creation modules are integrated. Once test 
items are entered during test creation, they 
are transferred and formatted into the 
Presentation Analysis module. 

Further Developments 

As the World-Wide-Web (WWW) 
expands, so does the potential to provide 
distance learning opportunities. While 
rapid, seemingly unprecedented 
developments occur with World-Wide- 
Web browsers such as Netscape, the 
interactivity required for computer-based 
testing is not yet available. However, it is 
inevitable that further developments will 
expand the degree to which users can 
interact with WWW documents. It is the 
author's plan to integrate the TestMaker 
modules with the WWW. For distance 
learning purposes, a web page can be set 



up so that instructors and students at any 
location will be able to access a form in 
which to enter test items. This information 
can then be sent to a server containing the 
Presentation Analysis module which will 
import and format these items for analysis 
and presentation to remote sites. For 
example, a teacher-education instructor 
teaching concepts of test construction to 
two remote sites could ask each site to 
work in groups to generate objective test 
items for a particular topic. Test items 
could be entered into the WWW form and 
sent to the Presentation Analysis module 
for importing and formatting. They could 
subsequently be presented to both sites for 
students to critique and analyze. 

Future developments with the 
TestMaker project will investigate the 
possibility of testing students at remote 
distance learning sites. While there are 
many issues which must be taken into 
account with remote testing (e.g., test 
security, cheating), it may provide 
worthwhile benefits. Test items could be 
posted on the World-Wide-Web. The 
Presentation Analysis module could 
receive students responses (e.g.. entered 
via the WWW) to questions. The 
instructor could import the data into the 
module and subsequently analyze it. At a 
minimum, this could reduce instructor time 
spent on grading, as well as provide for 
easy updating of test items. Also, 
compared to paper-and-pencil tests, this 
approach, given the analysis capabilities of 
the Presentation Analysis module, would 
provide for more thorough test reviews 
and analysis during class lectures. 

Summary 

lU recent years, it has become 
apparent that computers serve many 
functions and offer distinct advantages 
over some traditional educational 
assessment methods. In many instructional 
settings, computers manage the routine 
tasks related to assessment. They are 
capable of presenting test items and 
evaluating them. Computer-based tests, 
among other things, give educators the 
capability of providing immediate feedback 
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or knowledge of results to learners and 
interaction during testing. This paper 
discussed a project aimed at providing 
educators with a means by which to 
develop computer-based tests. The project 
investigates instructional strategies such as 
interaction during testing that may help 
students generate their own meaning from 
information and subsequently construct 
knowledge. Means to present and analyze 
test items for distance learning classes 
were also discussed. Currently, delivering 
objective tests over a distance is often 
difficult. While the approaches for 
distance learning testing presented in this 
paper do not resolve many of these issues, 
they do provide educators with some 
strategies for testing and learning. As 
developments in the World-Wide-Web 
continue, it may eventually be possible to 
provide effective testing methods and 
associated instructional strategies at a 
distance. 
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